Stable Diffusion
Contents
Stable Diffusion#
Note
Install ekorpkit package first.
Set logging level to Warning, if you don’t want to see verbose logging.
If you run this notebook in Colab, set Hardware accelerator to GPU.
!pip install ekorpkit[art] exit()
Prepare Environment#
%config InlineBackend.figure_format='retina'
%load_ext autotime
from ekorpkit import eKonf
eKonf.setLogger("INFO")
eKonf.set_cuda(device=0)
print("version:", eKonf.__version__)
is_colab = eKonf.is_colab()
print("is colab?", is_colab)
if is_colab:
eKonf.mount_google_drive()
project_dir = eKonf.set_workspace(workspace="/content/drive/MyDrive/workspace/", project="ekorpkit-book")
print("project_dir:", project_dir)
INFO:ekorpkit.base:Setting cuda device to ['A100-SXM4-40GB (id:0)']
INFO:ekorpkit.utils.notebook:Google Colab not detected.
INFO:ekorpkit.base:Setting EKORPKIT_WORKSPACE_ROOT to /content/drive/MyDrive/workspace/
INFO:ekorpkit.base:Setting EKORPKIT_PROJECT to ekorpkit-book
INFO:ekorpkit.base:Loaded .env from /workspace/projects/ekorpkit-book/config/.env
version: 0.1.40.post0.dev7
is colab? False
project_dir: /content/drive/MyDrive/workspace/projects/ekorpkit-book
time: 2.19 s (started: 2022-11-07 10:56:36 +00:00)
Create a stable diffusion instance#
# from ekorpkit.batch import BaseBatchRunConfig
# cfg = BaseBatchRunConfig(batch_name = "test", random_seed=True)
# cfg
time: 197 µs (started: 2022-11-07 10:56:38 +00:00)
from ekorpkit.models.art.stable import StableDiffusion
# cfg = eKonf.compose("model/stable_diffusion")
# sd = eKonf.instantiate(cfg)
sd = StableDiffusion()
INFO:absl:Unable to initialize backend 'tpu_driver': NOT_FOUND: Unable to find driver in registry given worker:
INFO:absl:Unable to initialize backend 'rocm': NOT_FOUND: Could not find registered platform with name: "rocm". Available platform names are: CUDA Host Interpreter
INFO:absl:Unable to initialize backend 'tpu': module 'jaxlib.xla_extension' has no attribute 'get_tpu_client'
INFO:ekorpkit.batch:Setting seed to 1543449010
INFO:ekorpkit.batch:Batch name: stable-batch, Batch num: 0
INFO:ekorpkit.batch:Setting seed to 2683389793
INFO:ekorpkit.batch:Batch name: stable-batch, Batch num: 0
INFO:ekorpkit.models.art.base:> downloading models...
INFO:ekorpkit.models.art.base:> loading modules...
INFO:ekorpkit.models.art.base:No modules to load
INFO:ekorpkit.models.art.base:> loading models...
INFO:ekorpkit.base:Loaded .env from /workspace/projects/ekorpkit-book/config/.env
INFO:ekorpkit.base:Loaded .env from /workspace/projects/ekorpkit-book/config/.env
time: 54.6 s (started: 2022-11-07 10:56:38 +00:00)
Batch Run#
prompts = ['examining documents, artstation trends beautiful digital anime painting of a happy cyborg child surrounded by bright cute cute little bubbles bursting out of the clouds with glowing lights, dramatic lighting, intricate, highly detailed, unreal engine, Trend',
'examining documents. There is also a detailed matte painting of a village in the middle. art station, concept art, matte, sharp focus, illustration.. by michael welan, raymond swanland, and greg rut',
'examining documents in an enchanted room surrounded by colorful glowing orbs, dark and ominous mood, volumetric lights, red and cyan theme, cinematic lighting, high quality, highly detailed, octane render, 8 k resolution, trending on artstation',
'examining documents. holographic. photorealistic. highly detailed. cinematic. 4 k closeup portrait shot of beautiful pale gothic girl in white dress and necklaces surrounded by waterfalls with glowing butterflies, intricate,',
'examining documents and videos, trending on artstation, 8k, HD portrait of danny devito smoking a cigar, intricate, elegant, glowing lights, highly detailed, digital painting, artstation, glamor pose,',
'examining documents. intricate elegant. Trending on artstation. 8k a hyper detailed matte painting of a city landscape inside of a gigantic blackberry bushes. at night. dense foliage. aerial photography. extreme bokeh.',
'examining documents and schematics, highly detailed, octane render, 8 k ultra realistic illustration of beautiful female japanese dragon princess, flowing hair, cyberpunk, intricate elegant ornate details, by alphonse',
'examining documents, hyper detailed, ultra realistic, concept art, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie',
'examining documents and detailed white background, elegant detailed digital painting, concept art, matte, sharp focus, 4k ultra hd, illustration, magical, unreal engine, octane render, cinematic lighting, wide angle establishing shot, extremely high detail',
'examining documents and document insert an emoji with or without spaces, in a cyberpunk city, by Guy Denning, Johannes Itten, Derek Gores, Russ Mills, glitch art, octane render, iridescent accents']
prompts
['examining documents, artstation trends beautiful digital anime painting of a happy cyborg child surrounded by bright cute cute little bubbles bursting out of the clouds with glowing lights, dramatic lighting, intricate, highly detailed, unreal engine, Trend',
'examining documents. There is also a detailed matte painting of a village in the middle. art station, concept art, matte, sharp focus, illustration.. by michael welan, raymond swanland, and greg rut',
'examining documents in an enchanted room surrounded by colorful glowing orbs, dark and ominous mood, volumetric lights, red and cyan theme, cinematic lighting, high quality, highly detailed, octane render, 8 k resolution, trending on artstation',
'examining documents. holographic. photorealistic. highly detailed. cinematic. 4 k closeup portrait shot of beautiful pale gothic girl in white dress and necklaces surrounded by waterfalls with glowing butterflies, intricate,',
'examining documents and videos, trending on artstation, 8k, HD portrait of danny devito smoking a cigar, intricate, elegant, glowing lights, highly detailed, digital painting, artstation, glamor pose,',
'examining documents. intricate elegant. Trending on artstation. 8k a hyper detailed matte painting of a city landscape inside of a gigantic blackberry bushes. at night. dense foliage. aerial photography. extreme bokeh.',
'examining documents and schematics, highly detailed, octane render, 8 k ultra realistic illustration of beautiful female japanese dragon princess, flowing hair, cyberpunk, intricate elegant ornate details, by alphonse',
'examining documents, hyper detailed, ultra realistic, concept art, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie',
'examining documents and detailed white background, elegant detailed digital painting, concept art, matte, sharp focus, 4k ultra hd, illustration, magical, unreal engine, octane render, cinematic lighting, wide angle establishing shot, extremely high detail',
'examining documents and document insert an emoji with or without spaces, in a cyberpunk city, by Guy Denning, Johannes Itten, Derek Gores, Russ Mills, glitch art, octane render, iridescent accents']
time: 6.39 ms (started: 2022-11-07 10:57:33 +00:00)
batch_name = "batch-run-test"
text_prompts = "Brave and beautiful new world in Jeju, artstation matte painting."
random_seed = False
batch_run_params = {
"text_prompts": prompts,
"guidance_scale": [5, 10, 15],
"steps": [50, 100, 250, 500],
}
batch_run_pairs = [
# ("guidance_scale", "steps"),
['text_prompts']
]
batch_results = sd.batch_imagine(
batch_name=batch_name,
batch_run_params=batch_run_params,
batch_run_pairs=batch_run_pairs,
random_seed=random_seed,
seed=123,
)
print(batch_results)
INFO:ekorpkit.batch:Using existing path: /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen
INFO:ekorpkit.batch:Setting seed to 2962579712
INFO:ekorpkit.batch:Batch name: batch-run-test_text_prompts, Batch num: 11
INFO:ekorpkit.batch:Merging config with args: {}
INFO:ekorpkit.batch:Setting seed to 2895784382
INFO:ekorpkit.batch:Batch name: batch-run-test_text_prompts, Batch num: 11
INFO:ekorpkit.models.art.base:Prompt: examining documents and document insert an emoji with or without spaces, in a cyberpunk city, by Guy Denning, Johannes Itten, Derek Gores, Russ Mills, glitch art, octane render, iridescent accents
INFO:ekorpkit.models.art.config:Saving batch run config to /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/outputs/stable-diffusion/batch-run-test/configs/batch-run-test_text_prompts(0)_run_configs.yaml
[PosixPath('/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/outputs/stable-diffusion/batch-run-test/configs/batch-run-test_text_prompts(0)_run_configs.yaml')]
time: 10min 34s (started: 2022-11-07 10:57:33 +00:00)
Generate images#
text_prompts = "a living tiger inside a gorgeous cave in a futuristic Jeju island, relaxing night, watercolour on the ceiling, glowing on glass, light art, dark, dark, gorgeous, dark, bright in an abandoned village in a futuristic alien nightscape"
batch_name = "cave-jeju"
time: 663 µs (started: 2022-11-07 09:13:57 +00:00)
eKonf.setLogger("INFO")
sd.verbose = True
text_prompts = "examining documents and detailed white background, elegant detailed digital painting, concept art, matte, sharp focus, 4k ultra hd, illustration, magical, unreal engine, octane render, cinematic lighting, wide angle establishing shot, extremely high detail"
batch_name = "lecture"
results = sd.imagine(
text_prompts,
batch_name=batch_name,
num_samples=4,
guidance_scale=9,
width=800,
height=400,
num_inference_steps=150,
)
INFO:ekorpkit.batch:Using existing path: /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen
INFO:ekorpkit.batch:Setting seed to 589320020
INFO:ekorpkit.batch:Batch name: lecture, Batch num: 6
INFO:ekorpkit.batch:Merging config with args: {}
INFO:ekorpkit.batch:Setting seed to 2315390883
INFO:ekorpkit.batch:Batch name: lecture, Batch num: 6
INFO:ekorpkit.models.art.base:Prompt: examining documents and detailed white background, elegant detailed digital painting, concept art, matte, sharp focus, 4k ultra hd, illustration, magical, unreal engine, octane render, cinematic lighting, wide angle establishing shot, extremely high detail
time: 1min (started: 2022-11-07 09:13:57 +00:00)
sd.config.imagine
{'mode': 'generate', 'text_prompts': 'examining documents and detailed white background, elegant detailed digital painting, concept art, matte, sharp focus, 4k ultra hd, illustration, magical, unreal engine, octane render, cinematic lighting, wide angle establishing shot, extremely high detail', 'num_samples': 4, 'num_images_per_prompt': 1, 'width': 800, 'height': 400, 'num_inference_steps': 150, 'guidance_scale': 9, 'init_image': None, 'mask_image': None, 'inpaint_strength': 7.5, 'seed': None, 'increase_seed_by': 1, 'display_image': True, 'save_image': True, 'save_image_config': True, 'make_collage': True, 'display_collage': True, 'save_collage': True}
time: 4.16 ms (started: 2022-11-07 09:18:33 +00:00)
Inpaint images#
base_url = "https://github.com/entelecheia/ekorpkit-book/raw/main/assets/figs"
init_image_path = f"{base_url}/chu-horse.png"
mask_image_path = f"{base_url}/chu-horse-masking-rough.png"
mask_image_exact_path = f"{base_url}/chu-horse-masking-exact.png"
init_image = eKonf.load_image(init_image_path)
mask_image = eKonf.load_image(mask_image_path)
mask_image_exact = eKonf.load_image(mask_image_exact_path)
sd.compare_images([mask_image, init_image, mask_image_exact], resize_ratio=0.5)
time: 2.53 s (started: 2022-11-03 10:57:07 +00:00)
prompt = "a golden statue of an eagle with clouds, colorful painting by Tanya Hern, mural on the roof, digital art, artwork, beautiful, colorful, visual art on landscape, surrealism, watercolor, vivid by stunning"
results = sd.imagine(
prompt,
mode="inpaint",
init_image=init_image_path,
mask_image=mask_image_path,
num_samples=4,
num_images_per_prompt=2,
inpaint_strength=10,
num_inference_steps=100,
return_including_init_image=False
)
INFO:ekorpkit.batch:Using existing path: /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen
INFO:ekorpkit.batch:Batch name: cave-jeju, Batch num: 1
INFO:ekorpkit.batch:Merging config with args: {}
INFO:ekorpkit.batch:Setting seed to 2776875379
INFO:ekorpkit.models.art.base:Prompt: a golden statue of an eagle with clouds, colorful painting by Tanya Hern, mural on the roof, digital art, artwork, beautiful, colorful, visual art on landscape, surrealism, watercolor, vivid by stunning
INFO:ekorpkit.batch:Saving config to /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/outputs/stable-diffusion/stable-batch/config/cave-jeju(1)_config.yaml
time: 54 s (started: 2022-11-03 10:57:10 +00:00)
Stitch images#
results = sd.imagine(
text_prompts,
mode="stitch",
batch_name=batch_name,
num_samples=6,
num_images_per_prompt=2,
guidance_scale=7,
inpaint_strength=10,
)
INFO:ekorpkit.utils.notebook:shell type: ZMQInteractiveShell
INFO:ekorpkit.models.art.stable: >> elapsed time to imagine: 0:02:49.117101
INFO:ekorpkit.batch:Saving config to /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/outputs/stable-diffusion/stable-batch/config/cave-jeju(2)_config.yaml
time: 2min 49s (started: 2022-11-03 10:58:04 +00:00)
To view the panorama, upload the image to renderstuff.
Show Configuration#
sd.show_config(batch_name=batch_name, batch_num=0)
INFO:ekorpkit.batch:Using existing path: /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen
INFO:ekorpkit.batch:Batch name: cave-jeju, Batch num: 0
INFO:ekorpkit.batch:Loading config from /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/outputs/stable-diffusion/stable-batch/config/cave-jeju(0)_config.yaml
INFO:ekorpkit.batch:Merging config with the loaded config
INFO:ekorpkit.batch:Merging config with args: {}
INFO:ekorpkit.batch:Setting seed to 1894816122
{'_target_': 'ekorpkit.models.art.stable.StableDiffusion',
'autoload': True,
'batch': {'batch_name': 'cave-jeju',
'batch_num': 0,
'resume_run': False,
'run_to_resume': 'latest',
'seed': 1894816122,
'set_seed': 'random_seed'},
'collage': {'bg_color': 'black', 'cols': -1, 'max_images': 6, 'padding': 5},
'device': 'cuda:7',
'hf_user_access_token': None,
'imagine': {'display_collage': True,
'display_image': True,
'guidance_scale': 7,
'height': 512,
'increase_seed_by': 1,
'init_image': None,
'inpaint_strength': 7.5,
'make_collage': True,
'mask_image': None,
'mode': 'generate',
'num_images_per_prompt': 1,
'num_inference_steps': 100,
'num_samples': 2,
'save_collage': True,
'save_image': True,
'save_image_config': True,
'seed': None,
'text_prompts': 'a living tiger inside a gorgeous cave in a '
'futuristic Jeju island, relaxing night, '
'watercolour on the ceiling, glowing on glass, '
'light art, dark, dark, gorgeous, dark, bright in '
'an abandoned village in a futuristic alien '
'nightscape',
'width': 768},
'model': {'generate': {'device': 'cuda:6',
'pretrained_model_name_or_path': 'CompVis/stable-diffusion-v1-4',
'revision': 'fp16',
'use_auth_token': True},
'inpaint': {'device': 'cuda:5',
'pretrained_model_name_or_path': 'runwayml/stable-diffusion-inpainting',
'revision': 'fp16',
'use_auth_token': True}},
'module': {'library_dir': PosixPath('/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/libs'),
'modules': None},
'name': 'stable-diffusion',
'path': {'batch': {'base_dir': PosixPath('/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/outputs/stable-diffusion/stable-batch'),
'name': 'stable-batch'},
'batch_configs_dir': PosixPath('/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/outputs/stable-diffusion/batch_configs'),
'batch_dir': PosixPath('/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/outputs/stable-diffusion/stable-batch'),
'batch_name': 'stable-batch',
'cache_dir': PosixPath('/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/cache'),
'library_dir': PosixPath('/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/libs'),
'model_dir': PosixPath('/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/models'),
'name': 'disco-imagen',
'output_dir': PosixPath('/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/outputs/stable-diffusion'),
'root': PosixPath('/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen'),
'tmp_dir': PosixPath('/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/tmp'),
'verbose': False},
'verbose': False}
time: 44.2 ms (started: 2022-11-03 11:00:53 +00:00)